Should we adjust for pupil background in school value-added models? A study of Progress 8 and school accountability in England

In the UK, US and elsewhere, school accountability systems increasingly compare schools using value-added measures of school performance derived from pupil scores in high-stakes standardised tests. Rather than naively comparing school average scores, which largely reflect school intake differences in prior attainment, these measures attempt to compare the average progress or improvement pupils make during a year or phase of schooling. Schools, however, also differ in terms of their pupil demographic and socioeconomic characteristics and these also predict why some schools subsequently score higher than others. Many therefore argue that value-added measures unadjusted for pupil background are biased in favour of schools with more 'educationally advantaged' intakes. But, others worry that adjusting for pupil background entrenches socioeconomic inequities and excuses low performing schools. In this article we explore these theoretical arguments and their practical importance in the context of the 'Progress 8' secondary school accountability system in England which has chosen to ignore pupil background. We reveal how the reported low or high performance of many schools changes dramatically once adjustments are made for pupil background and these changes also affect the reported differential performances of region and of different school types. We conclude that accountability systems which choose to ignore pupil background are likely to reward and punish the wrong schools and this will likely have detrimental effects on pupil learning. These findings, especially when coupled with more general concerns surrounding high-stakes testing and school value-added models, raise serious doubts about their use in school accountability systems

Comment: Citation Statistics

Comment on "Citation Statistics" [arXiv:0910.3529]Comment: Published in at http://dx.doi.org/10.1214/09-STS285C the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

The Limitations of Using School League Tables to Inform School Choice

In England, so-called ‘league tables’ based upon examination results and test scores are published annually, ostensibly to inform parental choice of secondary schools. A crucial limitation of these tables is that the most recent published information is based on the current performance of a cohort of pupils who entered secondary schools several years earlier, whereas for choosing a school it is the future performance of the current cohort that is of interest. We show that there is substantial uncertainty in predicting such future performance and that incorporating this uncertainty leads to a situation where only a handful of schools’ future performances can be separated from both the overall mean and from one another with an acceptable degree of precision. This suggests that school league tables, including value-added ones, have very little to offer as guides to school choice.Examination results, Institutional comparisons, League tables, Multilevel modelling, Performance indicators, Ranking, School choice, School effectiveness, Value-added

The effects of regional out-migration on job openings by occupation

This paper reports the results of two years of research on the estimation of regional occupational employment migration rates and their influence on estimates of future job openings by occupational group. The first section provides a general description of state and metropolitan area migration by occupation and other demographic variables. The descriptive statistics demonstrate that the overall level of out-migration rates and their variation among demographic and occupational groups are quite large relative to current employment estimates and estimates of future job openings that currently do not take into account regional out-migration. The second section describes the construction, and use of, estimated adjusted out-migration rates. The adjusted out-migration rates are created by using incomplete data methods to statistically combine data from the 1990 Census and the 1987 Occupational Mobility Current Population Survey. This hybrid data set contains complete information on occupational migration and mobility and allows us to isolate the out-migration rate that reflects changes in state-of-residence but not changes in occupation. These adjusted rates eliminate potential double counting that would be introduced using unadjusted occupational out-migration rates. This application of incomplete data methods is "tested" by generating a set of adjusted migration rates for a region of the United States and applying these rates to estimates of base year regional occupation employment to produce a set of estimated job openings that take into account regional out-migration. The results show that the total number of estimated job openings by occupation have to be revised significantly upwards when out-migration is taken into account.

Numerical indigestion: how much data is really good for us?

We are swimming in ‘big data’ and despite their performances as advocates of data freedom, policymakers don’t seem to bear any responsibility for educating the public on how to read it. Harvey Goldstein believes that academics must make it their mission to explain that evaluating statistical information is far from trivial

Modelling the Impact of Pupil Mobility on School Differences in Educational Achievement

The recently introduced National Pupil Database in England allows the tracking of every child through the compulsory phases of the state education system. The data from Key Stage 2 for three Local Education Authorities are studied, following cohorts of pupils through their schooling. The mobility of pupils among schools is studied in detail using multiple membership multilevel models that include prior achievement and other predictors and the results are compared with traditional ‘value added’ approaches that ignore pupil mobility. The analysis also includes a cross classification of junior and infant schools attended. The results suggest that some existing conclusions about schooling effects may need to be revised.Multilevel model, multiple membership model, mobility, value added, National Pupil database, educational attainment, cross classified model, random effects, school effectiveness